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Abstract 

The use of Web-based technology has enabled many government and corporate training divisions to 
reach more learners than ever before. Institutions are restructuring their budgets, obtaining funding 
from governments and foundations to fund resources needed to increase online learning offerings. 
While online learning is increasing, questions arise as to the quality when compared to traditional 
face-to-face instruction. Research indicates that online learning can be more effective than face-to- 
face instruction. The purpose of this project is to analyze training sessions used by a state judicial 
system in southeastern United States for their Involuntary Commitment (IC) training to determine if 
self-directed online training is a viable solution to replace the current face-to-face training program 
and if so, design, develop and evaluate a pilot program for online training material. 
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1. INTRODUCTION 

The use of Web-based technology has enabled 
many government and corporate training 
divisions to reach more learners than ever 
before (Bonk, 2002; Lynch, 2005; Smith, 2003). 
According to the 2010 State of the Industry 
Report, the American Society for Training and 
Development (ASTD) stated that in 2009; 
twenty eight percent of all training hours for 
organizations were completed online, up twenty 
three percent from 2008 (Prost, 2010). 
Institutions are restructuring their budgets or 
obtaining funding from governments or 
foundations to fund resources needed to 
increase online learning offerings (Lynch, 2005). 
While online learning is increasing, questions 
arise as to the quality when compared to 
traditional face-to-face instruction. 

The quality and benefits of face-to-face versus 
online instruction have been debated for years. 
The U. S. Department of Education sponsored a 
study which compared over one thousand 


research studies from 1996 through 2008 to 
measure the quality of face-to-face versus online 
instruction (Means, 2008). In his study in 2000, 
Johnson compared online instruction with face- 
to-face instruction for a variety of learning 
outcomes. He concluded that course quality was 
considered slightly more favorable in face-to- 
face instruction than in online instruction 
(Johnson, 2000). Training in the workplace has 
also been shown to be more clearly defined, 
more collaborative and more innovative with 
face-to-face instruction (Smith, 2008). Face- 
to-face instruction also provides more 
opportunities for instructors to tailor the training 
session according to the needs of the students 
during instruction (Jefferson, 2009). 
Nonetheless, face-to-face instruction has its own 
challenges to learning. 

Some studies suggest that if face-to-face 
training becomes instructor-centered it could 
encourage passive learning, ignore individual 
differences and needs of the learners (e.g., 
Banathy, 1994; Hannum & Briggs, 1982; 
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Johnson, 2000). Another limitation of face-to- 
face training is related to the degree of learner 
control. Learner control is where the learner 
believes their success is due to their own efforts 
(Lynch, 2004). Lynch (2004) contends that in 
face-to-face training sessions, learners may feel 
their success or failure is determined more by an 
external force and not by their own efforts. 
Flexibility of time and place are other obstacles 
for face-to-face learners. It is also argued that 
instructors often do not have enough time to 
adequately cover the material (Jefferson, 2009). 
Businesses usually allot only a small amount of 
time for training and expect learners to complete 
their learning while on the job. This can make 
mastering tasks difficult for learners who must 
perform immediately upon returning to work. 
Classrooms also have a limited number of seats 
and courses may not be offered when learners 
are available to attend. Face-to-face training 
can also be labor intensive and expensive 
(Bates, 2000). Organizations must pay the 
expense of travel for learners and instructors, 
equipment and instructional material costs. 
Thus, in spite of its many advantages, face-to- 
face instruction has limitations. Lack of learner 
control and inflexibility of time and place are 
some of the major limitations with face-to-face 
instruction. However, the question is can Online 
Learning eliminate some of these limitations and 
as a result, improve quality of instruction? 

Before we can determine if online learning is the 
answer for training organizations, we must first 
define online learning. Bates (2000) uses the 
University of North Carolina Institute of 
Academic Technology to define online or 
distributed learning as an environment with a 
learner centered approach to education. 
According to this definition distributed learning 
integrates technology to enable opportunities for 
synchronous and asynchronous activities, which 
gives instructors flexibility to customize learning 
environments to meet the needs of diverse 
student populations while providing high-quality 
and cost effective learning (Bates, 2000). 
Online Learning activities used in distributed 
learning are also defined as a pedagogy that 
emphasizes asynchronous small group 
discussions, collaborative problem solving, 
reflective inquiry, and competency-based 
outcomes (Rudestam, 2004). Parsad (2008) 
states that distance education includes 
instructional delivery of online courses on or off- 
campus, remote locations, correspondence 
courses and hybrid/blended online courses. To 
summarize, online learning promotes a learner 
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centered approach for synchronous or 
asynchronous learning activities and provides 
more opportunities for utilizing customized 
learning environments that can be accessed 
anywhere or anytime. For the purposes of this 
project we will focus primarily on asynchronous, 
self-directed online learning environments rather 
than synchronous learning. Now that we have 
defined online learning, the question remains, is 
online learning the answer to provide quality 
instruction for training organizations? 

Benjamin Bloom (1984) compared the results of 
one-on-one tutoring with face-to-face classroom 
instruction in his 2-Sigma problem. He found 
learners achieved up to two standard deviations 
higher in favor of one-on-one tutoring. Although 
we cannot provide an instructor for every 
student, educational technologies such as online 
training has the potential of individualizing 
learning and can make education both affordable 
and accessible (Fletcher, 2007). In another 
study, researchers completed meta-analysis on 
over one thousand empirical studies and 
determined that on average, online learners 
performed better than those receiving face-to- 
face instruction (Means, 2009). One reason for 
learner's high performance was due to flexibility 
of time and place. This flexibility gave learners 
access to additional learning time with the 
instructional materials. Learners also performed 
better when face-to-face and online instruction 
was blended together than with face-to-face 
instruction alone. Group participation in online 
learning enables the learning to be constructed 
through the interaction with others and the 
outcome is enhanced learner satisfaction (Bonk, 
2002; Smith, 2008). The quality implications 
presented here are just a few of the many 
advantages to online learning; however there 
have been challenges associated with online 
learning as well. 

Researchers point to a number of challenges 
regarding asynchronous online learning 
environments. Many of these challenges are 
associated with factors related to motivation and 
social presence or the degree to which people 
are perceived as "real" in computer-based 
communication (Gunawardena, 1995). For 
example, learners must be able to motivate 
themselves to learn on their own and teach 
themselves new information (Jefferson, 2009). 
There is also delayed feedback response time 
from both learners and instructors when asking 
questions or clarifying instructions. It is also 
difficult to form personal relationships both with 
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peers or instructors when courses are online 
only (Jefferson, 2009). Another study examining 
social presence in online courses found that 
students with high perceptions of social presence 
scored high in perceived learning and 
satisfaction just as students with low social 
presence scored lower in perceived learning and 
satisfaction (Richardson 2003). Even though 
motivation and social presence identified as 
challenges to online asynchronous learning, it is 
still considered a viable method of providing 
quality instruction for training organizations. 

In summation, online training can be perceived 
as effective; if not more effective, than that of 
face-to-face training. Johnson completed a 
study in 2000 that compared student satisfaction 
with their learning experience in both a face-to- 
face and online learning environment. Johnson 
(2000) found that there was no difference in the 
quality of the learning but that students found a 
slightly more positive experience with online 
instruction (Johnson, 2000). Although face-to- 
face training may still be more advantageous in 
many cases, online learning has become the 
wave of the future. Learning in the workplace is 
initiated by individuals as part of their working 
lives. It is informal, self-directed and broken 
into small chunks of learning. It is driven by 
short term needs not by any conscious plan of 
study (Bates, 2000). Although there are startup 
and maintenance costs associated with 
developing online training, it can also be less 
expensive than face-to-face training. Once the 
expense of development, installing equipment 
and networks are completed, online training can 
be conducted by one instructor to many learners 
both near and far (Bates, 2000). Travel 
expenses are greatly reduced as are the 
expense of printing instructional materials which 
can be delivered electronically (Strother, 2002). 
Learners can complete online training any time 
and any place either from their workplace or 
from the privacy of their own homes (Parsad, 
2008). Learners can also review the 
instructional materials many times to enhance 
their learning experience. In sum, online 
learning brings flexibility, of time and place, 
increased learner satisfaction and learner control 
and lower costs than with face-to-face 
instruction. 

2. PROPOSE OF THE STUDY 

The purpose of this project is to analyze training 
sessions used by a state judicial system in 
southeastern United States for their Involuntary 
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Commitment (IC) training to determine if self- 
directed online training is a viable solution to 
replace the current face-to-face training 
program and if so, design, develop and evaluate 
a pilot program for online training material. 

The Context of the Study 

The judicial system in southeastern United 
States is established as a co-equal branch of the 
state government. The local judicial agency 
within the state judicial system serves the 
technology, training and regulatory needs of the 
system. The local agency supports over six 
thousand five hundred employees in over 200 
offices throughout the state. The agency has a 
Legal Division, Technology Division, Human 
Resources Division and Training Division. The 
technology division provides employees with 
hardware and software resources. The training 
division works closely with the technology 
division to design, develop, implement and 
evaluate training for software solutions to 
employees. The agency has face-to-face, hands 
on computer training facilities in the corporate 
office. 

Traditionally, upon identification of the training 
needs, the agency would offer face-to-face 
classroom training. Face-to-face training 
sessions are also offered quarterly in the 
corporate training center. Previously, the IC 
application training has also been offered only 
for face-to-face, hands on classroom instruction. 
The face-to-face IC application training enabled 
users to utilize software application during the 
training to record specific data in a secure, 
centralized, electronic repository, efficient data 
sharing with the federal government as well as 
between different local offices. 

The development of the face-to-face application 
training required the coordinated efforts 
between the Technology Division (TSD), Training 
Division and external agencies. To meet the 
training needs for implementation deadline of 
the new Web-based application, the agency has 
recently begun to incorporate online learning 
into their face-to-face training course offerings. 
The online training, however, has been limited to 
synchronous meetings (webinars) using tools 
such as Cisco WebEx. In order to continue 
offering online courses, the agency has also 
improved its technology resources and provides 
access to computers to enable the organization 
and its staff to utilize online training. 
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3. NEEDS ANALYSIS 

Needs analysis was conducted to assess the 
effectiveness of the training for the application. 
Analysis of statistical reports generated by the 
agency and interviews with the project sponsor 
revealed approximately 20 percent of the offices 
are using the application. Analysis also revealed 
that those 20 percent enter 80 percent of the 
data into the application; the remaining 20 
percent of the data is entered by users who do 
not use the application on a daily basis. Further 
investigation revealed that there have been only 
three four-hour face-to-face training sessions 
held at the training center from January 1, 2009 
through December 31, 2010. Interviews with 
the management revealed that because of 
budget restraints and instructor availability, the 
number of training sessions was limited to only 
four sessions for a two year period. The 
sessions were also poorly attended because they 
were not flexible in time and place for learners. 
Learners were also informed that they would 
only be reimbursed for a portion of their travel 
expenses to attend the training. Analysis also 
revealed that only 15 learners attended the 
training sessions. A minimum of 200 users have 
access to the application and the ability to enter 
data on a daily basis. Only 7.5 percent of the 
users with access to the application have 
attended training sessions. Several application 
enhancement recommendations include making 
data entry into the application more efficient and 
incorporate user suggestions into the new design 
to promote "buy in" from the user group. 

Assessment of the effectiveness of synchronous 
online delivery of the training for the new Web- 
based application revealed that it has some of 
the same limitations as the face-to-face training. 
Travel expenses were reduced when compared 
to face-to-face training, but online attendance 
remained low and only represented by the few 
offices already using the application. The 
number of training sessions has also been 
limited due to budget constraints, travel 
restrictions and availability of the two instructors 
who teach the application. The synchronous 
online training sessions have been available for 
one year. A total of thirty six synchronous 
training sessions were held from February to 
October. The training sessions were two hours 
long and only offered on 12 days from February 
to October. There were ten sessions in 
February, four in March, eleven in June, five in 
July and six in October. Interviews revealed 
that scheduling training sessions was difficult 
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due to limited instructor availability. Training 
sessions were held within those few days of 
availability. The results of learner evaluations 
revealed that the learners had little choice of 
days or times when to enroll in the training 
sessions and commented they would like 
training to be more flexible to their schedules. 

A Summative evaluation was conducted for 
existing synchronous training sessions and found 
that learners' attitudes toward the training were 
mixed. Learners found the training organized 
and easy to follow, instruction was related to 
their work and instructional material was useful 
when returning to the job. Learners did not 
believe adequate time was allowed nor were 
there hands on activities included in the 
instruction. Interviews with the instructors 
revealed that they were only given two hours to 
complete each training session. Given the short 
amount of time involved, instructors did not 
have time to include hands on activities within 
their instruction. Instead, learners were given 
instructional materials which included examples 
and scenarios in the appendix. Learners also 
commented that once returning to work, they 
found they were unable to retain learned 
information and had to refer to the instructional 
materials. They also found the materials difficult 
to review on their own without the aid of the 
instructor. The instructional material was not 
designed to stand alone without learners 
attending training sessions. 

Analysis has shown that face-to-face and 
synchronous online training sessions has not 
solved the problems of flexibility of time and 
place, increased learner satisfaction and learner 
control. Synchronous online training has 
reduced travel expenses for learners but still 
include travel expenses for instructors and 
equipment expenses. As a result of the needs 
analysis, it is recommended that self-directed, 
online training would address the needs of the 
learners, support learner control and have 
flexibility of time and place. The development of 
self-directed learning materials will not be 
without cost. Flowever, the reduction in 
instructor and equipment expenses will 
compensate for some of the development cost. 
Furthermore, Self-directed asynchronous online 
training sessions would enable learners to have 
the flexibility to review and revisit the instruction 
as many times as needed to refresh their 
memories. Training would be designed to 
address the needs of the learners without the 
need for an instructor to be present. Learners 
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will be able to ask questions and receive 
feedback from instructors through asynchronous 
means. Learners will be required to register for 
the self-directed online training. Registration is 
required to ensure learners can access the host 
website and to establish communication between 
the instructor and learners. Once registered, 
learners be sent a welcome email which will 
identify the instructor's contact information, how 
to access the self-directed training and who to 
contact for technical support. Learners will have 
unlimited access to the self-directed material 
and a 24 hour helpdesk support. Learners will 
also be able to email instructors with questions 
about the training. By using asynchronous 
means of communication, instructors have the 
flexibility to answer questions and provide 
feedback from any location and whenever they 
have time. 

4. RECOMMEND SOLUTION 

Based upon the research, data analysis and 
interviews, the recommended solution is to 
design, develop and evaluate a pilot program for 
online, self-directed training materials for the IC 
application. All technology resources have been 
established for learners and online training is a 
viable option to use for IC application training. 
The design of the online training material will 
follow the conclusions found in the literature 
research. The instructional material will be self- 
directed and broken into chunks of learning, 
have flexibility of time and place and can be 
reviewed multiple times to reinforce learning. 
Travel expenses will be greatly reduced as will 
the expense of printing training materials. The 
training will be produced through using 
advanced multimedia technology tools and 
delivered through the agency Intranet. The 
training will address the needs of the learners, 
provide environment for practice, serve as 
refresher training without extra cost and effort 
and offer learner flexibility and control. 

In order to have flexibility of time and place, 
address the needs of the learners and support 
learner control, the self-directed, self-paced 
online training was developed with the following 
goals in mind. 

• The self-directed, self-paced training will be 
available to learners any time and from any 
place in order to achieve the learning 
objectives of the training session. 

• The learners will be able to retake (repeat) 
the course as needed. 
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• Learners will be able to transfer the 
knowledge and skills to their work 
environment upon completion of the self- 
directed training. 

5. LEARNING OUTCOMES 

Several learning outcomes were identified for 
the self-directed online training. First, Learners 
will be able to apply a set of rules to determine 
what is entered into the software application. 
Second, ensure the identified data is entered 
properly and without any error and lastly, self- 
assess the accuracy of the data that are entered 
into the system. Since only cases that contain 
a specific set of circumstances should be entered 
into the application, the learner is expected to 
identify which set of circumstances required data 
entry and which did not. 

The objectives were defined using a task 
analysis to identify high, medium and low level 
tasks. Once all the tasks were identified, the 
instructional goal, learning outcomes and 
learning objectives were developed and refined. 
The assessment strategies were based on the 
instructional goal and learning outcomes. The 
assessment items represent measurable 
concepts of the instructional goal and objectives. 
Once measurable concepts were identified, 
assessment items and instructional strategies 
were developed. 

6. INSTRUCTION DESIGN MODEL 

The instructional method used for designing the 
self-directed online instructional materials is the 
Learning by Doing or Goal-Based Scenario model 
developed by Roger C. Schank (Riegeluth, 
1999). Shank's model is based on skill 
development and learning factual information 
within the context of how it will be used. Shank 
based his model on several core values which 
include: 

• Learning to do skills, not just know factual 
information. 

• Learning occurs in the context of goals that 
are relevant, meaningful and interesting to 
the learner. 

• Knowledge learned is in terms of relevant 
tasks and how learners will use it outside the 
learning environment. 

The instructional material was designed to follow 
learning by doing or the goal-based scenario 
(GBS) training method. Using this model 
instruction was designed to incorporate Learning 
by Doing simulation where learners pursue goals 
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by practicing target skills and using relevant 
content to help them achieve their goals. 

Environmental Analysis 

Environmental Analysis showed that network 
infrastructure and computer hardware 
installation was completed and was available to 
all users throughout the state. The agency had 
set up a public website which could be accessed 
anywhere and contained published information 
for both employees and the public. A private 
computer network or Intranet has also been 
established with employee only access to all 
secured agency information and documentation. 
The self-directed training material was created 
and delivered individually for the initial pilot 
training. The Instructor and learners accessed 
training materials from their own computers. 
See Appendix B for a sample welcome letter that 
will be emailed to learners. 

Learner Analysis 

Learner analysis was also conducted to identify 
characteristics of the target audience. The 
analysis showed that the state agency had over 
six thousand users with access to various 
software applications within the agency. These 
users are current agency employees, both men 
and women, of many races and ethnicities with 
ages ranging from 18 to 73 years old. Learner 
education levels range from a minimum High 
School Diploma or GED through PHD. Learner 
abilities range from moderate to advanced level. 
The entry behaviors survey indicates learners 
have basic to advanced computer skills and 
moderate to advanced English reading and 
comprehension skills. Learners also had 
prerequisite knowledge needed to enter data 
into agency applications and discriminate case 
information. Learners may not have necessary 
prerequisite knowledge of the instructional 
delivery system and will be directed to view 
tutorials already available to them before taking 
the online learning materials. Learners believed 
the use of self-directed training could enable 
them learn at their own pace while still being 
able to perform job responsibilities in a timely 
fashion. Learners appeared to be highly 
motivated and wanted to acquire skills to protect 
their jobs while having satisfaction in performing 
well on the job. See Appendix A, Table 1 for 
more information about the target learners. 
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7. FORMATIVE EVALUATION PROCESS 
Methodology 

The formative evaluation was conducted using 
the Dick and Carey Criteria (Dick 2005). As 
suggested by Dick, Carey and Carey (2009), 
one-on-one and small group formative 
evaluation was conducted to assess the 
effectiveness of the module and to identify the 
areas for improvement and change. The ARCS 
Motivation Model and modules' learning 
outcomes were used as a framework to conduct 
both one-to-one and small group trials 
evaluation process. The Dick and Carey 
formative evaluation Criteria suggest assessing 
the clarity of instruction, the impact on learning 
and feasibility of time and resources. The clarity 
of instruction is how clear the message (content 
of the instruction), images, links and procedures 
are to the learner. The impact on learning deals 
with the learners attitudes toward the instruction 
and their achievement of the objectives. The 
feasibility considerations are the capability of the 
learner and the appropriateness of resources or 
environment. The ARCS Motivation Model 
suggests evaluating the appropriateness of 
instructional strategies to gain learners 
attention, provide relevant information, ensure 
learner confidence to succeed and satisfaction 
with the learning experience. Thus, the 
outcomes for one-to-one trials are to ensure the 
instruction contains appropriate vocabulary, 
complexity and examples for the learner, yields 
positive learner attitudes and achievement and 
is feasible and useable within the given 
resources and environment. The outcomes for 
the small group trial are to refine the instruction 
to maximize effectiveness for the target 
audience. 

Instruments 

The formative evaluation instruments used to 
collect data include a survey, informal 
observation, a pretest and a posttest. The 
pretest and post included five questions based 
upon the course objective of entering, assessing 
and correcting data within the software 
application. The questions addressed the 
learning objectives which were determining what 
data to enter, demonstrating data entry 
procedures, assessing accuracy of data entered 
and correcting errors. The posttest question 
items were parallel to the pretest question items 
in order to measure learners' achievement of 
learning objectives. The survey was divided into 
four sections. The first section included 
questions about the clarity of instruction, the 
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impact on the learner and feasibility. The 
second section included navigation, feedback 
and organization questions. The third section 
included questions about the overall features of 
the module, the quality of instruction, relevance, 
gaining and maintaining learners' attention, and 
satisfaction with the learning experience. The 
last section included open ended questions 
about what learners liked and disliked and 
suggestions for improvement. See Appendix C 
for the survey, pretest and posttest instruments. 

Participants 

The self directed instructional module was 
created because the target audience is located in 
various cities across the state. Due to time 
constraints to complete the formative 
evaluation, only six participants were available 
at the time of the evaluations. Three individuals 
were available in person and three live in remote 
parts of the state. The participants of the One- 
to-One evaluation and small group of the 
Involuntary Commitment Training Module were 
six adult learners. Three of the evaluations were 
conducted one-on-one and since the module was 
self directed three of the evaluations were 
conducted remotely. The six participants 
consisted of two individuals with no experience 
in the Involuntary Commitment subject matter, 
one individual with some experience and three 
individuals with more than five years experience. 
Three learners have advanced level computer 
skills with experience in various types of 
software applications and three learners have 
moderate level skills also experienced in various 
types of software applications. Learners ages 
ranges from 30 to 60 years old. All learners 
were contacted before they began the 
instruction to explain the procedure, how to 
access the components of the instruction and 
answer any questions. 

Procedure 

The one-to-one evaluations were conducted in 
person for all three evaluations. This was to 
ensure consistency in functionality learners. 
Learners were under observation as they 
completed the pretest, instructional module and 
posttest. The author answered questions and 
recorded their comments and suggestions. 
Learners were allowed to complete the survey 
on their own to increase their confidence and 
comfort level when rating the instructional 
materials. The remote evaluations were initiated 
through Skype with video access to the learners 
and the ability to share desktops was used for 
observation. For the remote evaluations, the 
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instructional module was loaded onto a web 
server and learners were given detailed 
instructions and links to the module, pretest, 
posttest and survey. Due to technical difficulties 
with software, the remote learners were unable 
to access the quiz questions imbedded within 
the instructional module. As an alternative 
solution, the quiz questions were emailed as an 
attached word document the learners completed 
while viewing the module. The learners then 
emailed the quiz with answers back to me after 
completing the instruction. The problem was 
fixed later, after the evaluations were complete. 
A thank you email was sent to the learners for 
their evaluations and included the new website 
link so learners could view the final product with 
quizzes in its entirety. 

8. EVALUATION RESULTS 

Overall learners achieved learning outcomes and 
commented that they did learn information they 
did not know before completing the instruction. 
All of the learners scored well on the 
instructional module quiz. Three learners 
answered all the questions correctly, two 
learners missed one question and one learner 
missed two questions. The incorrect answers 
were not concentrated within a single learning 
objective. The results of the pretest and 
posttest were also good. The results of the 
pretest were, four learners scored four out of 
five questions correct and two learners scored 
five out of five questions correct. A score of 
five out of five correct means that the learner 
would not necessarily have to complete the 
instructional module; however, this formative 
evaluation was conducted by three learners who 
are very familiar with the subject although they 
are not subject matter experts. On the posttest, 
two learners scored four out of five and four 
learners scored five out of five questions correct. 
Learners stated that incorrect answers were 
possibly due to the question instructions not 
being clear and the difficulty in using the quiz 
feature to answer questions correctly. See 
Appendix C, Table 2 and 3 for the quiz and 
pretest, posttest learning gain score results. 

The survey indicated that overall, the learners 
had positive perception about quality of the 
instruction. The learning material gained and 
maintained their attention and they were 
satisfied with the learning experience. Learners 
thought the material was relevant to their 
needs. Learners also agreed the purpose and 
goals were clearly stated, it was appropriate for 
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individuals with various levels of computer 
experience and it was organized and easy to 
navigate. All learners but one agreed the length 
of the module was appropriate. One learner 
thought the overall length of the module was 
long but stated they were glad it was broken up 
into short sections five minutes or less so they 
did not have to complete the module in one 
session. The primary negative issue was lack of 
quiz feedback within the module. The quiz 
components of the module were designed using 
a specific software quiz functions which limited 
the design capability. The module quiz design 
did provide feedback when working properly; 
however, due to technical difficulties the remote 
learners did not have the ability to view the quiz 
questions within the module after it was posted. 
The learners were given verbal feedback during 
the evaluations but stated that they would like 
to see the quiz function working within the 
module itself. See Appendix D, Table 4 for the 
survey results. 

9. CONCLUSION 

In conclusion, research revealed that while face- 
to-face training may still be more advantageous 
in many cases, online learning can be as 
effective if not more effective. Online learning 
has become the wave of the future. It brings 
flexibility, of time and place, increased learner 
satisfaction and learner control and once 
developmental costs are incurred, lower costs 
than with face-to-face instruction. The 
instructional module developed was self directed 
based on face-to-face instructional material. 
Instructional designers should realize they may 
spend just as much, if not more time designing 
the module than for a face to face instruction. 
The content had to be developed in its' entirety, 
knowing that an instructor would not be present 
while learners are completing the module. This 
created a different set of challenges than when 
designing face-to-face instruction. Further 
evaluation with a larger sample group is needed 
to determine more accurate learning outcomes. 
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Appendix A - Learner Analysis 

Table 1: Learner Analysis Matrix 


Information 

Categories 

Data Sources 

Learner Characteristics 

Entry Behaviors 

Interviews, Data 
analysis, Observation 

Learners have basic to advanced computer skills 
and moderate to advanced English reading and 
comprehension skills. 

Prior Knowledge 

Data analysis, 
Interviews 

Learners have prerequisite knowledge needed to 
enter data into agency applications and 
discriminate case information. 

Learners may not have prerequisite knowledge of 
the instructional delivery system and will be 
directed to view tutorials available to them before 
they beqin the traininq. 

Attitudes 

Interviews, 

Observation 

Learners believe the use of self-directed training 
will enable them learn at their own pace while still 
being able to perform job responsibilities in a 
timely fashion. 

Motivation for 

Instruction (ARCS) 

Interviews, 

Observation 

Learners are highly motivated. They want to 
acquire skills to protect their jobs and have 
satisfaction in performinq well on the job. 

Education and ability 
levels 

Data analysis 

Learners have a minimum High School Diploma or 
equivalent through PHD level of education. 

Learners have moderate to advanced learning 
abilities. 

Relevance of skills to 
workplace 

Interviews, Data 
analysis, Observation 

Learners view the skills learned in the training are 
directly related to their work environment and 
therefore relevant and valuable. 
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Appendix B - Welcome Letter 

The Welcome letter will be sent to learners via Email once they have registered for the module. 

Dear [Learner], 

Welcome to the Involuntary Commitment Application Training Module. I am [Instructor's 
Name] and will be the instructor for this course. My contact information is below. 

This instructional module contains information about how to use the application from the 
Intranet to enter data into the system. The module can be found on the company website, click on 
the link provided below to access the instructional module. 

The Instructional module includes both Audio and Video components and can be viewed on 
any computer that has access to the Website. The module is designed to take thirty minutes to 
complete and is divided into seven sections the Introduction, Classify Cases, The Main Menu, How to 
use Case Add, How to use Search, Verifying Reports and How to use Case Edit. Each section is 
approximately five minutes long. The recommended instructional flow is to begin at the Introduction 
and continue sequentially throughout the sections. 

There are no prerequisites to the course however each learner should have a userid and 
password to gain access to the Intranet application. Contact your supervisor to receive necessary 
forms and instructions. 

Please take a moment to complete the Pretest found in the link below before completing the 
instructional module. After completing the instruction, complete the Evaluation Survey and Posttest 
found in the links below. These evaluations enable me to determine if the instruction is effective and 
identifies areas that may need revised. 

Thank you for taking time from your busy schedule to complete this instructional module. If 
you have any comments or questions please contact me via email any time. I hope you find the 
instruction informative, helpful and enjoyable. 

Sincerely, 

[Instructor Name] 

Instructor 

Email: 

Phone: 

Click on the link below to access the instructional module or type the address into the web browser 
Click the link below to access the Module Evaluation Survey 
Click on the link below to access the pretest 
Click on the link below to access the posttest 
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Appendix C - Formative Evaluation 

Survey Instrument 

The link to the formative evaluation survey is: 


1. Involuntary Commitment Module Survey 

After completing the Instructional Module, please answer the following questions. 



1. Consider the following statements about the content of the instructional module and 
rate them on a scale of 1-5 where 1 indicates that you strongly disagree and 5 indicates 

that you strongly agree. 






5 Strongly Agree 

4 Agree 

3 Unsure 

2 Disagree 

1 Strongly Disagree 

The purpose and/or goals 

of the module are clearly 

stated. 

0 

o 

0 

0 

0 

The module is appropriate 

for individuals with various 

computer experience 

levels. 

0 

o 

o 

o 

0 

The information was 

presented in a manner that 
made it easy to understand. 

0 

o 

o 

o 

o 

The information presented 
was appropriate in length. 

Other (please specify) 

0 

o 

o 

o 

o 

1 






2. Consider the following statements about the navigation and function of the module 
and rate them on a scale of 1-5 where 1 indicates that you strongly disagree and 5 

indicates that you strongly agree. 






5 Strongly Agree 

4 Agree 

3 Unsure 

2 Disagree 

1 Strongly Disagree 

The module was organized 
and easy to follow. 

0 

o 

o 

o 

0 

The Table of Contents and 

software navigation were 

0 

o 

o 

o 

o 

available easy to operate 






The software provides 

feedback to user responses. 

0 

o 

o 

o 

o 

Other (please specify) 






1 _ 






3. Consider the following statements about the overall features of the module and rate 
them on a scale of 1-5 where 1 indicates that you strongly disagree and 5 indicates that 

you strongly agree 

5 Strongly Agree 

4 Agree 

3 Unsure 

2 Disagree 

1 Strongly Disagree 

The overall quality of Ihe 
instruction is good. 

0 

o 

o 

o 

0 

The material was relevant 

to my needs. 

0 

o 

o 

o 

o 

The module gained and 

maintained my attention. 

0 

0 

0 

0 

0 

1 am satisfied with the 

learning experience. 

Other (please specify) 

0 

o 

0 

0 

0 

1 
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4. What did you like best about the Involuntary Commitment instructional module? 

aJ 

5. What did you like least about the Involuntary Commitment instructional module? 

d 

d 

6. Please, suggest ways the Involuntary Commitment instructional module can be 
improved. 

d 

pretest 

The link to the Pretest is: 

Pretest - Involuntary Commitment Training Module 


Question 1 

What type of data should be entered into the application? 

A. Involuntary Commitment 

B. Voluntary Commitment 

C. Inpatient Commitment 

D. Substance Abuse Commitment 
Question 2 

Check the box next to the components of the Main Menu. 

Add Involuntary Commitment 
Edit Involuntary Commitment 
Search Involuntary Commitment 
View Reports 
Help 

Question 3 

Which of the data entry fields below are required to enter case information into the “Case Add” feature of the 
application? 

A. Name 

B. Address 

C. Social Security Number 

D. Judges’ Name 

Question 4 

What two data entry fields are used to search for information in the application? 

A. Name and address 

B. Social Security Number and Name 

C. File number and County 

D. None of the above 

Question 5 

What two items are analyzed to determine if there are any errors in data entry? 

A. The Case File and the Case Add screen 

B. The Judgment form and Data Entry Report 

C. The View Search screen and Case File 

D. None of the Above 
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Posttest 

The link to the Posttest is: 

Posttest - Involuntary Commitment Training Module 


Question 1 

What type of data should NOT be entered into the application? 

A. Involuntary Commitment 

B. Voluntary Commitment 

C. Inpatient Commitment 

D. Substance Abuse Commitment 

Question 2 

List the five components of the Main Menu in the Text Box below. 


Question 3 

Which of the data entry fields below are required to enter case information into the “Case Add” feature of the 
application? 

A. File number 

B. Address 

C. Identification Number 

D. All of the Above 

Question 4 

What feature is used to find case information that has been previously entered into the application? 

A. Add Involuntary Commitment 

B. Edit Involuntary Commitment 

C. Search Involuntary Commitment 

D. View Reports 

Question 5 

How are errors in data entry identified? 

A. By comparing the Case Add Screen to the Case File 

B. By comparing the Data Entry Screen to the View Report Screen 

C. By comparing the Data Entry Report to the Judgment Form 

D. By Comparing the Case File to the Case Report 


©2012 EDSIG (Education Special Interest Group of the AITP) 
www.aitp-edsig.org /www.isedj.org 


Page 31 





Information Systems Education Journal (ISEDJ) 


10(3) 
June 2012 


Appendix D - Evaluation Responses 

Table 2: Results of Quiz _ 


Learner 

Objective 1 
Classify 

Cases 

Objective 2 

Case Add 

Objective 2 
Search 

Objective 3 

View Reports 

Objective 4 
Case Edit 

Mastering 

Objectives 


01 

Q2 


01 

Q2 

03 

04 

01 

01 

Q2 

03 

04 

01 

# 

% 

Learner 1 

+ 

+ 


+ 

- 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

11 

92 

Learner 2 

+ 

- 


+ 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

11 

92 

Learner 3 

+ 

+ 


+ 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

12 

100 

Learner 4 

+ 

+ 


+ 

+ 

+ 

+ 

+ 

+ 

- 

+ 

- 

+ 

10 

84 

Learner 5 

+ 

+ 


+ 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

12 

100 

Learner 6 

+ 

+ 


+ 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

12 

100 

+ correct response 












- incorrect response 













Table 3: Learning Gain Score - Pretest and Posttest 



Learner 1 

Learner 2 

Learner 3 

Learner 4 

Learner 5 

Learner 6 

Pretest 

4 

5 

4 

4 

4 

4 

Posttest 

4 

4 

5 

5 

5 

5 

Learning Gain 

Score* 

0% 

-1% 

1% 

1% 

1% 

1% 


Table 4: Results of Survey 


Consider the following statements about the overall features of the module and rate them on a scale of 1 -5 where 1 
indicates that you strongly disagree and 5 indicate that you strongly agree. 

l=Strongly disagree; 2=Disagree; 3=Unsure; 4=Agree; 5=Strongly agree 

Question 

Learner 1 

Learner 2 

Learner 3 

Learner 4 

Learner 5 

Learner 6 

Consider the following statements about the content of the instructional module and rate them on a scale of 1 -5 where 

1 indicates that you strongly disagree and 5 indicate that you strongly agree. 

The purpose and/or 
goals of the module are 
clearly stated. 

5 

5 

5 

5 

5 

5 

The module is 
appropriate for 
individuals with various 
computer experience 
levels. 

5 

5 

5 

5 

5 

4 

The information was 
presented in a manner 
that made it easy to 
understand. 

5 

5 

5 

5 

5 

4 

The information 
presented was 
appropriate in length. 

5 

4 

5 

5 

3 

4 

Other (please specify) 







Consider the following statements about the navigation and function of the module and rate them on a scale of 1 -5 
where 1 indicates that you strongly disagree and 5 indicate that you strongly agree. 

The module was 
organized and easy to 
follow. 

5 

5 

5 

5 

5 

4 

The Table of Contents 
and software navigation 
were available easy to 
operate. 

5 

5 

5 

5 

5 

4 
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The software provides 
feedback to user 
responses. 

5 

5 

5 

3 

4 

3 

Other (please specify) 




Quiz not 
working 

No quiz 

No quiz 
pop ups 

Consider the following statements about the overall features of the module and rate them on a scale of 1 -5 where 1 
indicates that you strongly disagree and 5 indicate that you strongly agree. 

The overall quality of the 
instruction is good. 

5 

5 

5 

4 

4 

4 

The material was 
relevant to my needs. 

5 

4 

5 

3 

4 

5 

The module gained and 
maintained my attention. 

5 

4 

5 

5 

4 

4 

1 am satisfied with the 
learning experience. 

5 

4 

5 

5 

4 

4 

Other (please specify) 







What did you like best 
about the Involuntary 
Commitment 
instructional module? 

Easy to 
follow and 
understand 
the 

instructor 

It was easy 
to use and 
understand 
and 1 could 
do it at a 
time 

convenient 
for me 

there is not just 
one thing: (1) 
verbal and visual 
synchronization, 

(2) the ability to 
pause the 
video/presentation, 

(3) real life 
examples 

Very 

complicated 
topic was 
well 

presented 
and easy to 
follow. 

The 
module 
provided 
information 
that 1 didn't 
know about 
necessary 
data 

needed for 
court 

documents. 

Use of 
forms in 
instruction, 
cursor 
navigating 
to correct 
parts of 
form 

What did you like least 
about the Involuntary 
Commitment 
instructional module? 

No response 

No response 

1 am not sure there 
was anything 

Did not get 
to see if my 
answers to 
the 

questions 

were 

correct or 

not. 

That 1 had 
to go back 
into the 
module a 
couple of 
times to 
answer the 
questions, 
and then, 1 
wasn't able 
to answer 
them all. 
Also, 1 
didn't 
realize for 
quite 

awhile that 
by moving 
the mouse 
off of the 
module 
surface, 
the module 
surface 
was able to 
become 
enlarged. 

Some of 
the quiz 
answers 

were 

space 

specific. 

Example, 

one 

answer 
required a 
space 
after a 
comma, 

even 

though the 
content of 
the 

answer 

was 

correct. 

Please, suggest ways 
the Involuntary 
Commitment 
instructional module can 
be improved. 

No response 

No response 

The rate of speech 
could be slower 

This is a 
very 

informative 
module; 1 
did learn 

Clearer 
instructions 
as to what 
information 
to look for 

No 

response 
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the 

objectives 
without 
having any 
prior 

knowledge 
of the topic. 
The module 
did 

breakdown 
a very 
complicated 
topic 
making it 
easy to 
complete. 

to answer 
questions 
and to 
consider 
for giving 
feedback. 
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